Coupling and Heterogeneity in Ubiquitous Computing

As mobile computers’ processing and communications systems become more powerful, they can support interactive tools such as collaborative virtual environments. Similarly, mixed reality systems use some of the same technologies as ‘traditional’ collaborative virtual environments and virtual worlds, but they are increasingly coupled and interconnected with other media in a way that we usually associate with ubiquitous computing systems. The context of use of a system, and context as modelled within that system, may consist of a heterogeneous combination of both new and old media. This paper uses theoretical work on the interdependence and interpretation to discuss such coupling among heterogeneous media. Our long–term aim is better understanding of the design and use of such systems, and better design practice consistent with theory and studies of user experience. UBICOMP & EMBODIED INTERACTION A recent HCI text [6] drew upon philosophy in discussing the accommodation of new technology by users, and their appropriation of it as they find their own ways to use and understand it. Dourish suggested that everyday human interaction is embodied i.e. is non–rationalising, intersubjective and bodily activity. Traditional approaches to HCI offer many guidelines for system design, but do not take full account of embodiment, according to Dourish. They are not in accord with the activity they aim to support. He raises the issue of embodiment but draws back from offering specific principles and guidelines, favouring instead statements that help sensitise designers to the general issue, e.g. users, not designers, create and communicate meaning and users, not designers, manage coupling. This paper uses similar theory, but tries to move forward with regard to discussion and understanding of accommodation and appropriation, and design guidelines. It focuses on the details of how systems that involve a mix of media, such as ubicomp and mixed reality systems, are designed and used. It centres on the issue of heterogeneity—spatial, temporal and technological—as a catalyst of deeper understanding. Users of ubiquitous computing (ubicomp), mixed reality (MR) and augmented reality (AR) systems use the artifacts of digital media, such as handheld computers and headmounted displays, combined with artifacts in more traditional media, such as books, tabletops and buildings. In MR, AR and ubicomp, the distinction between digital media and traditional media is clear if one looks for it, but the idea is that, effectively, one is not aware of it because one focuses on the overall experience: on the task instead of the tools for the task, to put it crudely. The new technology and the seams where it joins to old media are, as Weiser put it [18], “literally visible, effectively invisible”. With such interwoven or simultaneous use, the notion of each medium being a space itself becomes problematic, as has been discussed in [3], [10] and [5]. Weiser suggested that even a “glass TTY UI can be ubicomp,” if its use is well woven into the fabric of people’s collaboration and interaction. This may seem contradictory to the common notion of ubicomp, involving technologies such as location sensors, mobile displays and wireless communication, but Weiser was clear that it was not technology in itself that made for ubicomp. Instead he suggested that we should aim for the accommodation and appropriation of computing into everyday life, so that its use is non–rationalized, intersubjective and interwoven with the other media that we use. In good design, according to Weiser, interaction using heterogeneous media is so tightly coupled in user activity that the obvious differences, boundaries and seams between the parts of a system become less significant than the quality of interaction with the whole. The seams are perceivable—the technology is ‘seamful’—but we can call the whole system a single, hybrid object because coupled use of the parts is so unproblematic in users’ interaction. In other words, interaction is non–rationalized and seamless, even if the technology is seamful. This approach to design brings to the fore the process of experiencing and understanding how to weave a new system into the other media used in one’s everyday life. It emphasises the temporal, spatial and social patterns of use of all the media one has at hand, rather than treating a tool or system as an isolated ‘thing in itself’. The ubicomp design approach relies on the fit and coupling of the system design with the context of use i.e. the full range of tools and media used in everyday communication, activity and interaction, and the social or cultural understandings of their use: “the unit of design should be social people, in their environment, plus your device” [18]. Social people, in their environment, continually mix and couple media in everyday communication—walking, gesturing and pointing while one talks, and referring to places and what people did in them as one writes—and computational media can or should become embedded and embodied in that mix and in that social interaction, and neither superior or inferior to more traditional media. People design their activity to fit ‘our’ technologies into the many media that they use in their everyday lives, often changing or adapting the technology along the way i.e. appropriating it to suit the practices and priorities of their own contexts and communities of use i.e. other, older tools and media, and their use in interaction with other people. Studies of use consistently point out that such accommodation and appropriation are key to the adoption of new technologies. This process has been observed in media spaces [7], email [13], Lotus Notes [15] and workflow technologies [2]. As people do this, the use of the new technology becomes everyday, in the sense that “the most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it” [17]. This notion of disappearance, where a tool is “literally visible, effectively invisible” is from philosophical hermeneutics [9,11]. An old example from Heidegger is the way that a skilled carpenter engaged in his work focuses on the use of the hammer, and how it changes and is combined with other tools and materials, rather than focusing on the hammer in itself. Heidegger called this practically engaged and non–rationalising use ‘ready–to–hand’, in contrast to the rationalising, objectifying and abstracting activity he categorized as ‘present–at–hand’. He saw both modes or categories of use as being set within a circular process of interpretation, in which one influenced by one’s understanding and past experience of older tools and media when using any new tool or medium. One’s use of the tool in the course of everyday, situated and social interaction, combining the new tool with the heterogeneous others used in everyday life, builds up new experience and understanding—that will affect how one uses and interprets another new tool. In time, this process of accommodation and appropriation lets one focus on the use of the tool, and not on the tool as a thing in itself, thus making the tool ‘disappear’. Influenced by Weiser but also drawing directly from similar philosophical sources, in [11] Dourish similarly called for a move towards design of interactive systems which have a better fit with everyday human activity, understanding and interaction, and with the practically engaged and non–rationalising way that everyday activity takes place. Dourish draws upon Heidegger, as well as Schutz’ elucidation of the social or intersubjective element of everyday perception and activity, Merleau–Ponty’s discussion of the way that the body, through the interwoven senses, plays a vital role in everyday perception, and Wittgenstein’s emphasis on the way that meaning and activity are based on the patterns of use of the heterogeneous mix of media that constitute language: “the meaning of a word is its use in the language”. Weiser and Dourish focus on raising our awareness of embodied interaction, i.e. the interpretation of a system by a user as ready–to–hand. They present traditional HCI design as being based on its opposite, i.e. rationalising, objectifying and abstracting activity, or interpretation by the user as present–at–hand. Dourish discusses the shift between these two categories of interpretation as varying the degree of coupling between the interpreter and the system. As he puts it [11, p. 139], the existence of both modes is critical to the effective use of technologies. However, Weiser and Dourish both swing from one extreme to the other, focusing almost entirely on design to support embodied or ready–to–hand interaction. They do not fully address the relationship between the two modes. In particular, how does a tool become invisible or ready–to–hand? Heidegger, and his successors such as Gadamer and Ricoeur, held that situations where a tool becomes present–at–hand may be crucial to the individual’s learning and to the differences between individuals. The ongoing ‘hermeneutic circle’ of interpretation and understanding integrates these two modes, and affords variation in people’s understanding as well as consistency in their behaviour. For example, creativity can be considered as the variation of an individual’s subjective understanding from his or her prior understanding and from others’. The individual may then be very conscious of his or her own activity, rationalising it and very aware of it, i.e. the system, tool or symbol is present–at–hand. With experience of its use, however, it may become understood and familiar, i.e. more ready–to–hand and embodied. Similarly, as two people perceive one another’s use, with each interpreting and reacting to each other, they can a c h i e v e i n t e r s u b j e c t i v e c o n s i s t e n c y of behaviour—consistent with each other, but not necessarily with the use expected by the designer. A use or activity that is new and present–at–hand for one of them can thus become learned and ready–to–hand for both. The circular process of interpretation, whereby perception and activity are influenced by understanding, but also feeding into and changing understanding, thus relies on the interplay between ready–to–hand and present–at–hand interpretation. Embodied interaction, as Dourish and Weiser made clear, is an aspect of human activity that is under–emphasised in HCI. Nevertheless, ready–to–hand embodied interaction and present–at–hand objectification are interdependent—and neither author addresses this. We have to expect that a new technology will be to some degree present–at–hand, no matter how well the designer aims towards embodied or present–at–hand interaction. This is most clearly the case when the technology is new, but two other situations arise that neither Weiser nor Dourish fully address. The first situation is breakdown, where the affordances of even the most familiar tool may significantly differ from those of everyday ready–to–hand use e.g. when the head of the carpenter’s hammer becomes loose, so that he has to consciously concentrate on using it towards his task. Another example might be the breakdown that occurs with a mobile phone when it loses its network signal: one’s attention may turn from a conversation ‘through’ the phone and its infrastructure to the tool itself. A second situation is where the task is the tool: where one can no longer work ‘through’ the tool in a transparent way because one has chosen to focus on the tool itself. This may occur as an act of conscious learning or analysis, e.g. a novice carpenter taking some time to improve his hammer swing, or a researcher studying how a new mobile technology works in use. Breakdown may have an influence, or overlap with this case, as one might be working to repair an earlier problem or to try a different tactic of use. HISTORY AND INTERDEPENDENCE IN MEDIA Activity continually combines and cuts across different media, building up the temporal patterns of coupling and interweaving that constitute experience and understanding. A person’s work or activity may be influenced by a 3D computer graphics display in front of them, and the interactions that such a system affords, but also by books, telephones, hypermedia, furniture, buildings and so forth—and other people’s use of all of these media. The context of one artifact, in a particular medium, is the other artifacts and tools in that medium—and also in the other media at hand. A narrow emphasis on one digital system or ‘virtual space’ as the paramount resource for activity underrates the interdependence of media. Recent technological developments, such as mobile phones and email, heighten or highlight the interdependence of media or intertextuality already familiar in the use of older media such as written text, maps and cinema, and well–explored in philosophy, semiotics [12, 27] and linguistics [31]. We take the standpoint, then, that activity and language is constituted by all the symbols and all the media one uses, with each symbol interpreted through immediate perception as well as past experience and social interaction. We can choose to characterise media and treat each one as if it were an isolated individuated entity because of the senses we use in perceiving each one, and also because of our understanding of how to relate and to distinguish examples of each one. The differences between media are usually, then, rather obvious. It is easy to distinguish the spoken word “red” from the written word red because of the senses one uses in each case. Despite having the same letters, it is easy to distinguish tar from rat by looking at the order of letters within each written word. Such simple rules about what one can immediately see, hear, etc. within a word begin to strain and then break when one considers, for example, how we distinguish homonyms such as rose. The written word rose can mean many things, including a flower and having risen. When spoken, the same syllables can also mean linear structures (rows), about or belonging to fish eggs (roe’s), moving in a boat (rows), small deer (roes) and multiple occurrences of the Greek letter (rhos). Saussure [16] established that a word’s usage is understood through understanding and experience of patterns of use i.e. of other symbols that generally co–occur with it in use in language—and not just through the perception of the word’s syllables or letters. Following Saussure and Wittgenstein [19], any symbol or artifact gains its meaning in this way, including a digital one: its meaning is its use in the language, where language is seem as involving all communicative media. The notion that context is the other symbols at hand, in all media, becomes progressively more important as we turn from thinking about the differences between media, and the distinction of artifacts or symbols in those media, and focus on the similarities of media and the relatedness of symbols. The meaning or understanding of each symbol is not solely dependent on its form or medium, but also on experience and understanding of how we use each symbol along with other symbols in any or all media. For example, the spoken word “red” and the written word red are related because, based on past experience and current context, we can use either of them in the context of rose blooms, fresh blood, the former USSR and so forth. We understand, relate and differentiate symbols through experience of combinations and patterns of use within a culture. We can more clearly see how one’s history has an effect on ongoing activity—Gadamer’s ‘historically effected consciousness’—in weaving media together via the hermeneutic circle. Overall we suggest that achieving the design goals of ubicomp and embodied interaction may be aided by understanding the interdependence of media, and supporting coupling in our system designs. The next section puts forward some more specific design examples and suggestions, intended to help towards this goal. COUPLING ACROSS MEDIA & TIME IN DESIGN A typical ‘context–aware’ ubicomp system involves the coupling and interdependence of media for an isolated user, and we often seem much keener to couple information to space than vice versa. A museum exhibition might be associated with a set of web pages, so that walking into a room on a particular architect triggers the display of text describing the life and work of that architect. However, reading text about the architect is less likely to trigger display of a map or visualisation of the museum room, and afford access to a structured collection of blueprints, design sketches and building models. We might not be surprised to see images from the museum via a webcam, but it is rare to find video going the other way, from the page reader back to the museum visitor. There are some partial counterexamples, of course, but we suggest that ubicomp systems are generally relatively asocial and asymmetric in terms of their coupling and use of media. Perhaps each medium should be coupled to the other, and part of the context of the other, so each space or text is a peer with others. In our work we aim to treat digital media more as peers, rather than treating any one space or tool as the primary focus or locus of activity. In our systems such as the Lighthouse system [4] and in ongoing work on a system called George Square, users interact with each other through audio links, as well as a number of spatial media. Users present themselves to others as icons on maps, as avatars in VRs and, of course, as people walking through buildings and city streets tracked by GPS. We couple spatial media together, tracking activity in each one and representing it in others, and we link the use of related artifacts in different media, such as using georeferenced web pages to show a location for someone accessing a web page, and to show a web page for a someone moving in a map, VR or city street. We note that greater symmetry does not mean absolute uniformity and homogeneity across media. Homogeneous shared resources may aid what Aoki et al. called a “cohesive social experience” [1], but slightly varied resources can serve as individual contributions and spurs to deeper engagement [4]. We suggest three practical situations in which a limited degree of heterogeneity may be useful: when users are in different locations or have different tools available and yet wish a shared experience; when the designer’s and the users’ interest is in the ambiguous or contradictory [8]; and when users have different past experiences to draw from. The latter point brings us to the way that ubicomp often focuses on context as based on immediately observable objective features, in a rather present–at–hand way, but context also has temporal and intersubjective features that cross or interrelate media—and that these features are especially important in ready–to–hand use. Again, there are some partial counter-examples in the literature, but we suggest that have a long way to go in making past activity across many media a resource for ongoing or synchronous activity in each medium and with each other. In George Square, we therefore have begun to support asynchronous awareness as well as the synchronous awareness of the Lighthouse system. In a way based on structuralist linguistics, we record user activity over time, so that we can make recommendations of where to go and what to read based on comparing each user’s recent activity with the past activity of others. We present each individual’s recommendations to each member of his or her group, as an aid to mutual awareness, but recommendations are heterogeneous with regard to users, and with regard to media: they may come from street movement, web activity, map activity or VR activity, or a mixture of the four. CONCLUSIONThis paper has drawn on hermeneutics and semiotics indiscussing the way that a narrow design focus on one spaceor medium as primary may inhibit use and constrainubicomp design, as everyday activity involves theinterweaving and combination of media. Similarly, weraised the issue of the relative lack of historicity ofubicomp systems. As Dourish put it [11], users, notdesigners, create and communicate meaning, and users, notdesigners, manage coupling—but designers are involved inthis process, and can support it with rich cross-mediaawareness between users, in both synchronous andasynchronous forms. By seeing the use of a computer system as one case of theuse or interpretation of a symbol, we bring to bear thephilosophy of language and interpretation, such asWittgenstein and Heidegger, and linguistics and semiotics,such as Saussure. Such discourse has already had asignificant effect in ubicomp and HCI, most obviously inthe work of Mark Weiser but, by raising the criticalawareness of the references and assumptions of Weiser andrelated researchers, we hope to enable future advance insystem design in terms of practice, theory, and accordbetween the two.REFERENCES1. Aoki, P. et al., Sotto Voce: Exploring the Interplay ofConversation and Mobile Audio Spaces, Proc ACMCHI, pp. 431–438, 2002. 2. Bowers, J et al. Workflow from Within and Without:Technology and Cooperative Work on the PrintIndustry Shopfloor, Proc. Euro. Conf. on ComputerSupported Cooperative Work (ECSCW), 51-66 1995. 3. Brown, B. & Perry M. Of Maps and Guidebooks:Designing Geographical Technologies, ACM DIS, pp.246–254, 2002. 4 . Brown, B. et al. Lessons from the Lighthouse:Collaboration in a Shared Mixed Reality System. ProcACM CHI 2003, pp. 577–584. 5 . Chalmers M (2003) Awareness, Representation andInterpretation, Computer Supported Cooperative Work11:389–409. 6. Dourish, P. Where the Action Is: The Foundations ofEmbodied Interaction. MIT Press, 2001. 7. Dourish, P. et al. Your place or mine? Learning fromlong-term use of audio-video communication,Computer Supported Cooperative Work 5(1), 33-62,1996. 8. Gaver, W. et al. Ambiguity as a Resource for Design,Proc. ACM CHI, pp. 233–240, 2003. 9. Grondin J. Introduction to Philosophical Hermeneutics.Trans. J. Weinsheimer. Yale U. Press, 1994. 10.Harrison S. & Dourish P. Re-Place-ing Space: TheRoles of Place and Space in Collaborative Systems,Proc. ACM CSCW, 67–76, 1996. 11. Heidegger, M. Being and Time, Harper Collins, 1962. 12. Kristeva, J. Word, Dialogue, and the Novel. In: T. Moi(Ed.), The Kristeva Reader, Columbia U. Press, pp.35-61, 1986. 1 3 .Mackay, W. Patterns of Sharing CustomizableSoftware, Proc. ACM CSCW, pp. 209-221, 1990. 14.Nöth W. Handbook of Semiotics. Indiana U. Press,1995. 15. Orlikowski, W. Learning from Notes: OrganizationalIssues in Groupware Implementation, Proc. ACMCSCW'92, pp.362-369, 1992. 16. de Saussure, F. Course in General Linguistics. Trans.Wade Baskin. McGraw–Hill, 1983 (Original: 1906.) 1 7 .Weiser, M. The Computer for the Twenty-FirstCentury, Scientific American, 94-110, Sept. 1991. 18.Weiser M. Creating the invisible interface: (invitedtalk). ACM Conf on User Interface Software andTechnology (UIST94), p.1, 1994. 19. Wittgenstein, L. Philosophical Investigations (3rd ed.),trans. G.E.M. Anscombe, Oxford U. Press, 1958